Skip to content

fix(tiles): bound a pack scan's record buffer by the file length - #4

Merged
HaoZeke merged 2 commits into
mainfrom
fix/pack-scan-bound
Nov 16, 2026
Merged

HaoZeke merged 2 commits into
mainfrom
fix/pack-scan-bound

Conversation

@HaoZeke

@HaoZeke HaoZeke commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Opening or refreshing a PackStore now stops at a record head that declares more bytes than the file holds, before sizing a buffer for it.

  • scan read a record head and resized its buffer to the path and stored lengths the head declared before reading them, so a crash that left a head and nothing after it cost whatever the torn bytes claimed, up to 4 GiB per open.
  • scan now takes the file length when it starts and stops at a head whose declared record would run past it, before the buffer is resized. A scan runs under the change lock whenever a writer truncates after it, so nothing appended meanwhile is mistaken for a torn tail.
  • The test appends a put head declaring 256 MiB after a good record and counts allocations while open_read scans it: the peak stays under 1 MiB.

The branch has a towncrier fragment.

scan reads a record head and resizes its buffer to the path and stored
lengths the head declares before reading them, so a crash that leaves a
head and nothing after it costs whatever the torn bytes claim, up to
4 GiB per open. The test appends a put head declaring 256 MiB after a
good record and counts allocations through a global allocator while
open_read scans it: the peak is the declared size where it should stay
under 1 MiB.
scan takes the file length when it starts and stops at a head whose
declared record would run past it, before the buffer is resized, so a
torn head is the end of the valid records it already was without the
allocation its lengths claim. A scan runs under the change lock
whenever a writer truncates after it, so nothing appended meanwhile is
mistaken for a torn tail; a reader picks such records up on its next
refresh as before.
@HaoZeke
HaoZeke merged commit 9479098 into main Nov 16, 2026
6 checks passed
@HaoZeke
HaoZeke deleted the fix/pack-scan-bound branch November 16, 2026 05:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant